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© Apparatus and method for disambiguating an Input stream generated by a stylus-based user 



© An apparatus and method in a tablet computer 
for disambiguating a handwriting input stream gen- 
erated by a stylus-based user interface. A tablet 
appBcaBon program interface (TAPI) of the present 
invention receives stroke data generated by a stroke 
made by a user on a digitizing tablet The TAPI 
inserts the stroke data into a database. If the stroke 
is an editing gesture, the TAPI routes the stroke to a 

N gesture recognizer which identifies an editing dialog. 

^ The TAPI then executes the dialog to perform oper- 
ations on the database. If the application program is 

Jjj expecting text input, the TAR continues to receive 

00 stroke data until it detects a closure event The TAPI 
then forms a stroke group entry in the database by 
grouping all non-gesture stroke data received since 

If) the last closure event The TAPI routes the stroke 



group data to a c 
candidate characters. The TAPI inserts the candidate 
characters into the stroke group entry. The TAPI 
notifies the appEcation program each time it updates 
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Technical Field 

The present invention relates generally to a 
method and apparatus for entering data into a 
computer system. More particularly, the present 
invention relates to a method and apparatus for 
entering data into a computer system through a 
stylus-based user interface. 

Background Art 

Computer systems which accept data streams 
generated by operating a stylus are becoming 
commonplace. A stylus based user interface gen- 
erally comprises a pen (called a stylus) and a 
digitizing tablet. The use of stylus-based user inter- 
faces is being driven by the general trend toward 
more natural user interfaces utilizing handwriting 
and freehand drawing. 

In many application programs, stylus-based 
user interfaces are superior to keyboards as a 
means for entering data. Such is the case, for 
instance, when a user of the computer has only 
one hand available for data entry. Other cases 
include, but are not limited to. those in which a 
keyboard would add too much bulk or weight to a 
data processing system designed to be highly por- 
table or the case of a system designed for opera- 
tion by a user who does not know how to or is 
physically unable to type. 

ThB difficulty in producing computer systems 
with stylus- based user interfaces has limited their 
marketability. Compared to an input data stream 
from a keyboard or mouse, an input stream from a 
stylus-based user interface is more difficult for the 
system to interpret and makes the development of 
stylus-based application programs very complex. 
The input stream of a keyboard or mouse 
(generally) unambiguously reflects a user's inten- 
tion, thatis. to select a particular keyboard key or 
mouse button. The application program may or 
may not be able to respond meaningfully to that 
particular input data, but the input data itself is 
clear. 

The stylus-based user interface, on the other 
hand, functions as a source of both character data 
(such as text, function keys and editing commands) 
and gesture data (j.e.. mouse data such as pointing 
and selecting). Regardless of which of these two 
types of data the user intends to enter, the input 
data stream of the stylus-based user interface con- 
sists of a series of x-y coordinate pairs (called x-y 
data) which describe the locus of the stylus as the 
user operates it Stroke data are generated from 
the x-y data. Stroke data represent strokes, which 
are sequences of the x-y coordinate pairs delimited 
by a pen-down event and a pen-up event A pen- 
down event occurs when the stylus first touches 



the digitizing pad. A pen-up event occurs when the 
stylus next leaves the digitizing pad. 

Some application programs, such as a sketch- 
ing application or a signature capture application 

s program, need to deal directly with these strokes. 
Most application programs, however, need only 
deal with the character data represented by the 
strokes. Accordingly, stroke data must be inter- 
preted before an application program can process 

io ft The interpretation may involve classifying the 
stroke data as gesture or character data, and/or 
converting them to character data. Accordingly, in- 
terpreting stroke data is a complex process which 
generally requires much programmer time to im- 

is ptement and much CPU to run. 

Several conventional approaches are employed 
to interpret stroke data. In a first such approach the 
application program interprets the stroke data. The 
application program receives stroke data, classifies 

20 them, and performs any necessary conversion to 
character data. A disadvantage of the first approach 
is that it requires the application programmer to 
extend considerable effort to team about stroke-to- 
keyboard data conversion and to write source code 

25 to implement the same. Furthermore, application 
programs generally have very specific ways of 
handling stroke data. As a result little of the con- 
version source code can be reused in other ap- 

M In a second approach, an operating system 
interprets the stroke data and passes the results to 
the application program. The Go Pen-point is an 
example of a computer system which employs the 
second approach. A major drawback of the second 

35 approach is that it is only feasible for operating 
systems designed to process stroke data. Accord- 
ingly, it cannot be used on such widely used op- 
erating systems such as DOS, Unix and OS/2. 
Therefore, what is needed is a system for 

« disambiguating an input stream from a stylus- 
based user interface which can be used by mul- 
tiple application programs on existing, widely used 
operating systems. 

45 Disclosure of Invention 

Accordingly, the present invention is directed, 
toward an apparatus and method in a tablet com- 
puter for disambiguating a handwriting input stream 

so generated by a stylus-based user interface. A tab- 
let application program interface (TAPI) of the 
present invention receives stroke data generated 
by a stroke made by a user on a digitizing tablet. 
The TAPI inserts the stroke data into a database, 

55 and informs the application program to which the 
stroke was directed that the database has been 
updated. The application program may then re- 
trieve the stroke data from the database. 
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If the stroke is an editing gesture, the TAPI 
routes the stroke to a gesture recognizer which 
matches the stroke data to a gesture prototype 
stroke. The gesture recognizer identifies an editing 
dialog associated with the prototype gesture stroke. 
The TAPI then executes the dialog to perform 
operations on the database, and notifies the ap- 
plication program that the database has been up- 
dated. The TAPI may also cause a display on a 
display device of the tablet computer to be redrawn 
to reflect the updated database. 

If the application program is expecting text 
input, the TAPI continues to receive stroke data 
until it detects a closure event delimiting a group of 
strokes. At that time, the TAPI forms a stroke group 
entry in the database by grouping all non-gesture 
stroke data received since the last closure event 
The TAPI routes the stroke group data to a char- 
acter recognizer which matches it to a number of 
character prototype strokes. 

The character recognizer then generates result 
data comprising the characters to which the char- 
acter prototype strokes are mapped and an indica- 
tion of the quality of each match. The TAPI inserts 
the result data into the stroke group entry and 
notifies the application program that the database 
has been updated. The TAPI may also cause the 
display on the display device to be redrawn to 
reflect the updated database. 

The foregoing and other objects, features and 
advantages of the invention will be apparent from 
the following more particular description of pre- 
ferred embodiments of the invention, as illustrated 
in the accompanying drawings. 

Brief Description of Drawings 

FIGURE 1 

is a block diagram of the hardware and software 
environment of the present invention; 
FIGURE 2 

is a block diagram of the TAPI of FIGURE 1 ; 
FIGURES 3A-3E 

are a flowchart of the operation of the present 

invention; 

FIGURE 4 

is a flowchart of the operation of a deletion 
dialog of FIGURE 2; 
HGURES 5A and SB 

show a flowchart of the operation of an insertion 
dialog of FIGURE 2; FIGURE 6 is a block dia- 
gram of a database of FIGURE 2; 
FIGURES 7A.7B and 7C 
are block diagrams of structures of an active list 
of FIGURE 6; 
FIGURE 6 

is a block diagram of a dialog module of FIG- 
URE 2; and 



FIGURE 9 

is a flowchart of the method for creating a dialog 
of the dialog module of FIGURE 8. 

s Best Modes for Carrying Out the Invention 

Hardware and Software Environment 



FIGURE 1 is a block diagram of a tablet com- 
jo puter 100 which uses a tablet application program 
interface (TAPI) 110 of the present invention. The 
tablet computer 100 comprises a hardware sub- 
system 112 and a software subsystem 114. The 
hardware subsystem 112 comprises a digitizing 
75 tablet 116 and a display device 118. The software 
environment 114 comprises the TAPI 110, a tablet 
device driver 120. a window system 122, a recog- 
nizer module 124 (in some implementations), an 
application program 126, and at least one applica- 
nt tion window 138. The window system 122 com- 
prises an window server 134 and PLI extensions 
136. The recognizer module 124 comprises a char- 
acter recognizer 140. character prototype data 142. 
and possibly a gesture recognizer 144 and gesture 
ss prototype data 146. 

A user 128 enters data into the tablet computer 
100 by operating a stylus on the digitizing tablet 
116. The digitizing tablet 116 generates x-y data 
representative of strokes made with the stylus. 
30 The tablet device driver 122 handles commu- 
nications between the digitizing tablet 116 and the 
software subsystem 114. For example, it initializes 
the digitising tablet 116, maintains a command 
dialog between it and an operating system, cap- 
os tures the x-y data from the digitizing tablet 116, 
transmits the x-y data (via the operating system) to 
the window system 122, and inks an inking plane 
130 of the display device 118. (Note that the inking 
of the inking plane 130 could instead be performed 
4a by a hardware interface of the digitizing tablet 
116.) 

The display device 118 receives data from the 
tablet device driver 120 (via its inking plane 130) 
and the window system 122 (via one or more 

4s application planes 132). The separate inking and 
application planes 130 and 132 facilitate distin- 
guishing between display data from the digitizing 
tablet 116 and display data from the window sys- 
tem 122. The distinction enables the tablet device 

so driver 120 to immediately ink stroke data. The 
stroke data can later be erased without affecting 
data written by the window system 122. Alter- 
natively, the window system 122 can make the 
displayed stroke data permanent by copying it to 

ss the application bit planes 132. In a preferred em- 
bodiment, the display device 118 is a conventional 
VGA display with four application planes 132 and a 
single inking plane 130. 
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Functions of the window system 122 include 
accepting input data from the operating system, 
managing input event queues and routing input 
events to the appropriate application window 13a 
The window system 122 could be a conventional 
window system such as the X Window System™, 
phis extensions to support stroke data. The X Win- 
dow System is available from the Massachusetts 
Institute of Technology. The extensions could be 
the Paper-Uke Interface (PU) extensions 136 to the 
X Window System, which were developed by the 
International Business Machines Corporation. 
These extensions are distributed as part of release 
5 of version X 11 of the X Window System. The 
PU extensions 136 comprise modifications to a 
window server 134 (such as that of the X Window 
System) to enable it to support additional event 
queues (including a stroke event queue) and to 
handle stroke data. 

Upon initialization of the application program 
126. each application window 138 associated with it 
registers with the window system 122 and can 
request to be notified of stroke events. Depending 
on the routing algorithm used by the window server 
134. when a stroke event occurs, it may be pre- 
sented to application windows 138 which include its 
x-y coordinates, to all of the application windows 
138, or according to other policies. Upon accepting 
the stroke event, the application window 138 also 
requests the actual stroke data. The stroke data are 
not used directly in the notification because, in 
general, the x-y data associated with each stroke 
are larger than the message size of the window 
system 122. After retrieving the stroke data, the 
application window 138 notifies the window system 
122 to delete the stroke data from its cache, and it 
notifies the tablet device driver 120 to erase the 
stroke from the inking plane 130. 

Depending on its nature, the application pro- 
gram 126 may write the stroke data to the applica- 
tion planes 132. ft may also pass the stroke data to 
the character recognizer 140 for conversion into an 
ASCII code (for handwriting input) or to the gesture 
recognizer 144 for conversion into a function code 
(for a gesture). Recognizers perform recognition by 
comparing strokes to examples in prototype data 
sets. Accordingly, a character prototype data set 
142 contains a collection of hand written examples 
of each valid character and a gesture prototype 
data set 146 contains a collection of handwritten 
examples of each valid gesture. 

Overview of TAP! 

The TAPI 110 comprises a set of modules and 
data structures for the management of stroke data 
The TAPI is architected as a programming inter- 
face and a set of data structures. In a preferred 



embodiment, the window system 122 provides the 
user-interface framework for the application pro- 

The application program 126 can configure the 

s TAPI 110 to operate in many different ways. For 
example, in a first configuration, the TAPI 110 
simply captures and displays strokes, with any 
other processing being performed by the applica- 
tion program 128 itself. Such a configuration might 

m be employed, for example by an application pro- 
gram 126 for sketching or drawing or by an ap- 
plication program 126 for signature capture. 

In a second configuration, the TAPI 110 cap- 
tures and displays strokes and performs hand- 
re writing recognition in a writing window which pops 
up on the display device 118 when the stylus first 
touches the writing window. 

In a third configuration, the TAPI 110 captures 
and displays strokes and performs handwriting reo 

so ognftion in an unlined. scrolling window on the 
display device 118 with in-situ editing and error 
correction invoked by appropriate gestures on rec- 
ognized text The editing and error correction ac- 
tions are determined by dialog procedures which 

25 use database primitives 224 (see FIGURE 2) of the 
database 212 to manipulate the stroke and char- 
acter data in the database 212. 

FIGURE 2 is a high level block diagram of the 
structure of a preferred embodiment of the TAPI 

30 110 of the present invention. In this embodiment, 
the TAPI 110 comprises an Xlib/Xtools widget set 
214, a writingA widget 210, a writingR widget 218, 
a writingD widget 220. a dialog module 222. a 
database 212, and a reco link 216. FIGURE 2 also 

35 shows how the TAP1 110 interacts with the applica- 
tion program 126 and recogniser 124 of FIGURE 1. 

A specific instance of the writing widgets 210, 
218 and 220 would generally be associated with 
each application window 138. Also, to facilitate por- 

40 lability, the writingA widget 210 contains all of the 
features of the TAP1 110 which are specific to any 
particular window system and widget set 

A widget is a re-usable object which manages 
an application window 138 presented on the dis- 

45 play device 118. It provides a path between user 
input events and the input mechanisms of the 
application program 126 that respond to these 
events. It also provides a path between the output 
mechanisms of the application program 126 and a 

so window manager of the display device 1 18. A wid- 
get may also provide specialized services to trans- 
form or otherwise respond to input or output 
events. 

Several standard widget sets have been devel- 
55 oped, including the Xlib/Xtools widget set 214, the 
Open Software Foundation's Motif widget set and 
Carnegie Mellon University's Andrew widget set. 
The user-interface portions of application program 
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126 written for X Windows environments are gen- 
erally constructed from a standard widget set and 
additional widgets to perform specialized functions. 
The additional widgets are written by the developer 
of the application program 126 but descend from s 
(i.e., inherit attributes from) widgets in the standard 
widget set. The additional widgets in the present 
invention are the writingA widget 210. the writingR 
widget 218 and the writingD widget 220. 

The writingA widget 210 descends from the io 
Motif drawingArea widget In addition to handling x- 
y data (as the drawingArea widget does) the 
writingA widget 210 provides an interface to the 
database 212. It also keeps track of strokes that 
occur between closure events, thereby facilitating is 
character recognition. 

The writingR widget 218 serves as an interface 
to the reco link, performs simple operations on the 
database 212 and supports character display. The 
writingD widget 220 serves as an interface to the so 
dialog module 222. Additional functions of the writ- 
ing widgets 210. 218 and 220 are described below 
in the section entitled "Operation of the TAPIV 

The dialog module 222 supports a set of 
database operations invoked by gestures made by 25 
the user 12a The database 212 comprises struc- 
tures for storing stroke data and recognition data, 
as well as primitives for modifying the data, writing 
the data to a file and reading the data from a file. 
The reco link 216 provides a configuration interface 30 
to the character recogniser 140 and the gesture 
recognizer 144. 

Database 

FIGURE 6 shows a block diagram of the struc- 
ture of the database 212. Looking at FIGURE 6. the 
database 212 comprises an active list 610. an 
inactive list 612 and database primitives 614. The 
database primitives 614 include a dump primitive 40 
616. a restore primitive 61ft a hit test primitive 620, 
a group strokes primitive 622. a create word entry 
primitive 624 and an add character primitive 626. 
as well as many others. The active list 610 contains 
data the user 128 has entered and not deleted. « 
while the inactive list 612 contains data the user 
128 has entered and subsequently deleted. 

FIGURES 7A, 7B, and 7C show the structure of 
the active fist 610. The structure of the active list 
610 is the same as that of the inactive list 612. so 
Accordingly. FIGURES 7A. 7B and 7C also illus- 
trate the structure of the inactive list 612. Looking 
at FIGURE 7A, the active list 610 comprises a 
stroke element list 710 (see FIGURE 7A). a stroke 
group list 712 (see FIGURE 7B). and a word list 55 
744 (see FIGURE 7C). Each stroke element entry 
714 in the stroke element list 710 comprises a 
header 716 and positional data 718. The header 



716 contains a window id 718 as well as other data 
such as the time at which a stroke was made. The 
window id 719 indicates the application window 138 
in which the stroke was made. The positional data 
718 contains the stroke's relative coordinates on 
the display device 118. 

The stroke group list 712 can have two kinds of 
entries. First, an unrecognized stroke group entry 
720 comprises a window id 722 and stroke element 
pointers 724. Each of the stroke element pointers 
724 points to a stroke element entry 714 asso- 
ciated with strokes which have been grouped to- 
gether, in the manner explained below in the sec- 
tion entitled "Operation of the TAPI*. 

Second, a recognized stroke group entry 726 
comprises the window id 722, at least one stroke 
element entry 714 and a result structure 72a The 
result structure 728 comprises at least one result 
730 and. with each result 730. a stroke elements 
used field 732. a score field 734 and a bit field 736. 
The stroke elements used field 732 indicates all of 
the stroke elements used by the recognizer 124 in 
determining the result 730. The score field 734 
indicates the certainty with which the recognizer 
124 determined the result 730. The bit field 736 
indicates other information about the result 730. 
such as whether or not it should be displayed and 
whether or not it is an active result The active 
result is initially the result 130 with the lowest score 
field 734 (and thus the greatest certainty). 

The result structure 728 further comprises an 
aggregate score 738 and a word separation field 
740. The aggregate score 738 indicates an estima- 
tion by the recognition module 124 of the probabil- 
ity that the result structure 728 contains a correct 
result The word separation field 740 indicates 
whether or not the distance between the strokes 
used in generating the result structure 728 and 
those used in generating the preceding adjacent 
result structure 728 exceed a predetermined 
threshold. 

Looking at FIGURE 7C, the word list 744 con- 
tains a number of word list entries 752. A word list 
entry represents a group of characters. La., a word. 
Specifically, the word list structure 752 has a win- 
dow ID 754 and a number of character pointers 
756. Each character pointer 756 points to one of 
thB recognized stroke group entries 726. 

Operation of TAPI 

Initial Processing of Stroke Data 



FIGURES 3A through 3E show a flowchart of 
the operation of the third embodiment of the invert- 
Hon. Looking at FIGURE 3A, in the step 310 the 
Xtib/Xtools widget set 214 configures the logical 
display of each application window 138. Application 
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window configuration involves specifying the posi- 
tion, shape and its type font of the window, the 
type of data It accepts (e.g., text, strokes or some 
combination of the two), and several other param- 
eters. 

In a step 312. the tablet device driver 120 
captures x-y data from the digitizing tablet 116. In 
the step 314. the tablet device driver 120 displays 
the x-y data by inking the inking plane 130 of the 
display device 118. 

In the step 316, the tablet device driver 120 
generates from the x-y data stroke events and 
stroke element entries 714. Generation of a stroke 
event involves generating a pen-down, pen-up or 
intermediate stroke event The tablet device driver 
120 generates a pen-down stroke event upon re- 
ceiving a code from the digitizing tablet 116 which 
indicates that the user 128 has placed the stylus on 
the digitizing tablet 116. Similarly, the tablet device 
driver 120 generates a pen-up event upon receiv- 
ing a code from the digitizing tablet 116 which 
indicates that the user 128 has lifted the stylus 
from the digitizing tablet 116. The tablet device 
driver 120 generates intermediate stroke events at 
regular intervals to indicate the location of the 
stylus. For example, it could generate an intermedi- 
ate event for every one hundred x-y coordinates 
received. The intermediate stroke events enable 
the TAP! 110 or the application program 126 to 
determine when the user 128 has finished moving 
the stylus and is just holding it steady on the 
digitizing tablet 116. 

Generation of the stroke element entry 714 is 
as follows. The tablet device driver 120 converts x- 
y coordinates generated between a pen-down and 
pen-up event to relative coordinates of the display 
device 118. and inserts the latter coordinates into 
the field for positional data 718 in the stroke ele- 
ment entry 714. The tablet device driver 120 then 
inserts additional information regarding the stroke 
(including the window id 719) into the header 716 
of the stroke element entry 714. 

In a step 318 the tablet device driver 120 
routes each stroke element entry 714 to the ap- 
plication window 138 through the window system 
122. Specifically, the tablet device driver 120 sends 
the stroke element entry 714 and the stroke events 
to the window system 122. The stroke element 
entry 714 is added to a stroke data cache of the 
PLI extension 136, and the stroke events are 
placed on a stroke event queue of the PU exten- 
sion 136. The window system 122 then routes the 
stroke events to the application windows 138 

Various policies may be used to determine 
which of the application windows 138 will be of- 
fered a stroke event For example, the stroke event 
might be offered initially to the application window 
138 it accepted the previous stroke event Or the 



stroke event might be offered to the application 
window 138 only if the associated stroke was made 
through it Or the stroke event might be offered to 
the application window 138 only if the application 

s window 138 is exposed. If the application window 
138 was offered the stroke event it responds to the 
window system 122 by indicating whether or not It 
will accept the stroke element entry 714 associated 
with the stroke event Generally, the application 

io window 138 will only be able to accept the stroke 
events if application window 138 was built with the 
TAP1 110 widgets 210. 218 and 220. 

In a step 320. the window system 122 deter- 
mines whether the application window 138 has 

»s accepted the stroke event K not then the steps 
322-328 are carried out as follows. In a step 322, 
the window system 122 converts the stroke event 
to a mouse event In a step 324. the tablet device 
driver 320 erases the x-y data in the inking plane 

so 130. In a step 326, the window system 122 routes 
the mouse event to the application window 138. 
Specifically, the mouse event is put on a mouse 
event queue of the window server 1 34. The window 
server 134 then routes the mouse event to the 

as application window 138. In a step 328. the applica- 
tion program 126 associated with the application 
window 138 which accepted the mouse event pro- 
cesses the mouse data. Row of control of the TAPI 
110 then returns to the step 312. 

30 . If, on the other hand, the application window 
138 did accept the stroke element entry 714, then 
the operation of the TAPI 110 is as follows. In a 
step 330, the writingA widget 210 requests the 
stroke element entry 714 from the stroke data 

35 cache of the PU extension 136 and transmits it to 
the database 212. When it does so ; the widgetA 
inserts an identifier of the application window 138 
(the window id 719) into the header 714. In a step 
332, the insert stroke primitive 621 of FIGURE 6 

M creates a new entry in the stroke element list 710 
of the database 212 and copies the stroke element 
entry 714 to it 

The writingA widget 210 then transmits an ap- 
propriate message through the window system 122 

« to the tablet device driver 120. The tablet device 
driver 120 responds, in a step 334, by erasing the 
x-y data of the stroke from the inking plane 130. 

Looking at FIGURE 3B, in a step 336 the 
writingA widget 210 determines whether stroke dis- 

so play is enabled for the application window 138 by 
consulting configuration parameters of the applica- 
tion window 138. If stroke display was enabled, 
then in a step 338 the writingA widget 210 inks the 
positional data 718 in the application planes 132 so 

ss that the stroke is displayed in the application win- 
dow 13a Otherwise, or after the step 338, in step 
340 the writingA widget 210 commands the window 
system 122 to delete the stroke element entry 714 
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from the stroke data cache of the PU extension 
136. 

In a step 342. the writingR widget 218 and the 
writingD widget 220 determine whether the dialog 
module 222 or the recognizer module 124 is en- 
abled for the application window 138. If not, then 
execution transfers immediately to a step 366 of 
FIGURE 3D, in which the vrritingA widget 210 noti- 
fies the application window 138 that there is a new 
stroke element entry 714 in the stroke element list 
710. If the application window 138 does not accept 
the entry, then flow of control returns to the step 
312 to capture additional stroke data (see step 
367). Otherwise, the application program 126 pro- 
cesses the stroke element entry 714 as follows. 

Processing of Stroke Data by Application Program 



In a step 388. the application program 126 may 
retrieve the positional data 718 from the stroke 
element entry 714. The application program t26 
then processes the positional data 718. as indi- 
cated by a step 370. In a step 372. the application 
program 126 determines whether a request has 
been issued for it to suspend execution. If not 
execution returns to the step 312 to capture addi- 
tional stroke data. 

Otherwise, in a step 373 the application pro- 
gram 128 saves the state of the database 212. It 
does so by invoking the dump primitive 816 of 
FIGURE 6 to write the data contained in the 
database 212 to a database file. The application 
program 126 then enters a suspended mode in a 
step 374. in which it continually tests to see if it 
has been commanded to resume. Upon receipt of 
the resume command, the application program 126 
restores the state of the database 212 in a step 
376. It does so by invoking the restore primitive 
618 to retrieve the data of the database 212 from 
the database file. Execution then returns to the step 
312. 

Processing a Gesture 



ff on the other hand, the dialog module 222 or 
the recognizer 124 was found to be enabled in the 
step 342, execution continues as follows. In a step 
344 FIGURE 38 the writingR widget 218 deter- 
mines whether to assume the stroke is a gesture. It 
could do so by. for example, assuming that the 
stroke is a gesture if an imaginary box drawn 
around the stroke would exceed a specified size. 

If the stroke is assumed to be a gesture, then 
execution is as follows. In a step 378 of FIGURE 
3E. the writingR widget 218 routes the stroke 
through the reco link 216 to the gesture recognizer 
144. Specifically, the writingR widget 218 routes to 
the reco link 216 a pointer to the associated stroke 



element entry 714. The reco link 216 responds by 
routing to the gesture recognizer 144 the stroke 
element pointer as well as a pointer to an anchor 
point in a heap. (A heap is a continuous portion in 

s the memory of the tablet computer 100.) 

The gesture recognizer 144 then performs a 
recognition algorithm by comparing the stroke to 
the sample gestures in the gesture prototype data 
set 146. The gesture recognizer 144 either in- 
fo dicates in the heap a dialog identification code 
associated with the matching sample or (if it found 
no match) terminates with an error code. Upon 
completion, it notifies the reco Dnk 216. 

in a step 380. the reco link 216 performs any 

76 necessary formatting of the data in the heap to 
make the result comprehensible to the writingD 
widget 220. Whether or how the data must be 
formatted depends on the nature of the gesture 
recognizer 144. 

so In a step 382. the gesture stroke is erased from 
the application window 138. In a step 384. the TAPI 
1 10 determines whether the gesture recogniser 144 
terminated with an error code. If so. the assumption 
in the step 344 was incorrect. The stroke was likely 

25 part of a character. Accordingly, execution returns 
to the step 312 to capture additional x-y data. 

ff. on the other hand, the result returned by the 
gesture recognizer 144 did indicate a valid gesture, 
then in a step 386 the writingD widget 220 deter- 

30 mines whether one of the dialogs is already active, 
ff none is active, then the gesture must indicate the 
desire of the user 126 to start one of the dialogs in 
the dialog module 222. Accordingly, in a step 388. 
the writingD widget 220 activates the dialog in- 

» dicated by the dialog ioentrfication code result 

If there was an active dialog when the step 386 
was executed, or after activation of the dialog in the 
step 388. then in a step 390 the active dialog 
executes any stroke data received since it last 

40 executed. Dialog execution is described below in 
the section entitled "Dialogs". 

In a step 392. the TAP1 1 10 determines wheth- 
er the active dialog has completed execution, ff so. 
in a step 394 the active dialog is deactivated. After 

45 the step 394 or a negative determination in the 
step 392, execution returns to the step 312 to 
capture additional stroke data. 

If. on the other hand, the stroke was not as- 
sumed to be a gesture in the step 344. then in a 

so step 346 the writingD widget 220 determines 
whether there is an active dialog. If so, then the 
stroke will be processed by the active dialog. Ac- 
cordingly, execution transfers to the active dialog at 
the step 390 of FIGURE 3E. which is explained 

55 above. 
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Response to a Closure Event 



If in the step 346 there was no active dialog, 
then in a step 348 the writingR widget 218 tests for 
a closure event A closure event delimits a com- 
plete handwritten symbol. It may be a hard event 
such as the user 128 touching a button icon on the 
display device 118, or it may be soft event such as 
a timeout following the end of a stroke or a rela- 
tively large movement of the stylus between the 
end of one stroke and the beginning of the next If 
there is no closure event, flow of control returns to 
the step 312 to collect additional stroke data. Oth- 
erwise, the operation of the TAP1 1 10 is as follows. 

In a step 350, the writingR widget 218 com- 
mands the database 212 to form a group of all the 
non-gesture strokes received since the previous 
closure event. The group primitive 622 (see FIG- 
URE 6) of the database 212 responds by: (1) 
creating an unrecognized stroke group entry 720 
on the active list 610 (see FIGURE 7), (2> setting 
the stroke element pointers 724 to point to the 
appropriate stroke element entries 718. and (3) 
copying the window id 719 of the stroke element 
714 to the window id 722 of the unrecognized 
stroke group entry 720. 

In a step 352. the writingR widget 218 routes 
the stroke group through the reco link 216 to the 
character recognizer 140. Specifically, the writingR 
widget 218 routes to the reco link 216 a pointer to 
the unrecognized stroke group entry 720. The reco 
link 216 responds by routing to the character re- 
cognizer 140 the stroke group pointer as well as a 
pointer to an anchor point in a heap. 

The character recognizer 140 then performs a 
recognition algorithm by comparing the stroke 
group to the sample stroke groups in the character 
prototype data set 142. The character recognizer 
140 indicates in the heap a number of result char- 
acters associated with matching samples, as well 
as. for each result character, the strokes actually 
used in the prototype comparison, a score (Ke.. the 
quality of the match) and other data. Upon comple- 
tion, the character recogniser 140 notifies the reco 
link 216. 

In a step 354. the reco link 216 performs any 
necessary formatting of the data in the heap to 
make the result comprehensible to the writingR 
widget 218. Whether or how the data must be 
formatted depends on the nature of the character 
recognizer 140. 

In a step 356. the writingR widget 218 attaches 
a result structure 728 to the unrecognized stroke 
group entry 720, thereby converting the latter into a 
recognized stroke group entry 726. Specifically, the 
writingR widget invokes the database primitives 
614 to copy the result data from the heap to the 
result structure 728. The writingR widget 218 then 
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computes the value of the word separation field 
740 and inserts this into the result structure 728. 
The word separation field 740 Indicates whether or 
not the distance between the strokes used in gen- 

5 eratirtg this result structure 728 and those used in 
generating the preceding adjacent result structure 
728 exceeds a predetermined threshold. 

In a step 357. the writingR widget 218 deter- 
mines whether the result just recognized should be 

io grouped with an existing word list entry 752. That 
is, it determines whether the word separation field 
740 indicates the threshold was exceeded. If so, 
then in a step 358, the writingR widget 218 com- 
mands the create word entry primitive 626 of the 

7s database 212 to create a new word structure 752 in 
the word fist structure 744 (see FIGURE 7C). Oth- 
erwise, or after the step 358. in a step 359 the 
writingR widget 218 commands the add character 
primitive 626 to add the result structure 726 to the 

» current word structure 752. 

In a step 360. the writJngA widget 210 deter- 
mines from the bit field 736 whether this result 
should be displayed and whether the application 
window 138 is enabled for display of the recogni- 

25 tjon result H both conditions are met then in a 
step 362 the writingA widget 214 erases the posi- 
tional data 718 inked in the application planes 132 
(in the step 338). In a step 364, the writingA widget 
214 inks in the application planes 132 the active 

so result 730 of the result structure 728 using the font 
defined by the configuration data of the application 
window 138. 

After the step 364 or a negative determination 
in the step 360. execution goes to the steps 366- 

35 376, which are described above. 

Dialogs 

Dialogs are incorporated within the writJngD 

« widget 220. but are implemented in a rule-based 
language so as to enable developers of application 
programs to easily modify and extend them. FIG- 
URE 8 shows a block diagram of the structure of 
the dialog module 222. The dialog module 222 

45 comprises intrinsic dialogs 810, external dialogs 
812 and a dialog manager 814. 

Intrinsic dialogs 810 are either fundamental ac- 
tions (such as stroke display) or complex actions 
(such as various kinds of error correction). The 

so intrinsic dialogs 810 include basic editing functions 
such as character deletion, character insertion and 
character replacement The intrinsic dialogs 810 
also include basic forms of error correction such 
as: (1) modifying the bit field 736 of the result 

55 structure 720 to reflect a different active result 730, 
(2) replacing a stroke with its recognized result, 
and (3) displaying a soft keyboard). The intrinsic 
dialogs 810 shown in FIGURE 8 are a delete char- 
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acter dialog 624 and an insert character dialog 826. 

External dialogs 812 are similar to an editor or 
spreadsheet macro. They can be created, modified 
and deleted by a software application developer. 
The external dialogs 812 use rules to define trans- s 
actions on the database 212. The external dialogs 
812 include the various responses to handwritten 
input reaching the end of the application window 
138. These responses include scrolling the input to 
the left, dynamically extending the application win- to 
dow 138, and allowing the user 128 to continue in 
the adjacent application window 138. The external 
dialogs 812 also include complex editing functions, 
such as flowing input to form justified paragraphs, 
and cut and paste movement of input among ap- rs 
plication windows 138 The external dialogs 812 
shown in FIGURE 8 are a delete word dialog 828. 
an insert word dialog 830 and an un-do dialog 832. 

The external dialogs 812 are written in a pro- 
gramming language which may be uniquely de- so 
fined for this purpose or which may be an existing 
language such as Prolog or C. Each external dialog 
812 is specified as a set of "if-then" rules in which 
the actions are governed by stroke data input by 
the user 128 and by results obtained from oper- 2s 
afJons on the database 212. The external dialogs 
812 are contained in a cKaJog file which is loaded at 
the time the application program 126 is initialized. 
The external dialogs 812 may then be compiled as 
part of the initialization, or may be interpreted when 30 
activated. 

The dialog manager 814 is an engine for ex- 
ecuting or interpreting the dialogs. 

FIGURE 4 is a flowchart which illustrates the 
operation of the delete character dialog 824 of 35 
deleting a single character which is displayed in 
either stroke or recognized form. The gesture 
which caused this intrinsic dialog to be activated (in 
the step 388 of FIGURE 3E) could be. for example, 
a vertical stroke drawn through the character or 40 
stroke group to be deleted. Looking at FIGURE 4. 
in a step 410 the writjngD widget 220 commands 
the database 212 to determine the character or 
strokes intercepted by the delete gesture. The hit 
test primitive 620 of the database 212 then locates 45 
the entry on the stroke group 1st 712 whose posi- 
tional data 718 corresponds to the display coordi- 
nates of the gesture. 

In a step 412, the writjngD widget 220 com- 
mands the database 212 to deactivate the located so 
entry by removing It from the active list 610. The 
entry is not destroyed, but is retained on the inac- 
tive list 612 in case the user 128 subsequently 
invokes the un-do dialog 828 to restore the deleted 
character. ss 

In a step 414. the writingfl widget 218 invokes 
the database primitives 614 to adjust the positional 
data 718 of any remaining unrecognized stroke 



group entries 720 (for a stroke group) or recog- 
nized stroke group entries 726 (for a character) on 
the active list 610 so as to over-write the deleted 
character. The database 212 then calls back the 
writjngA widget 210 in step 416 to cause the 
strokes or characters on the active list 610 to be 
re-displayed without the deleted character. The de- 
lete character dialog 824 is then deactivated, as 
shown in a step 418. The method of the delete 
word dialog 830 is simitar, except that it is invoked 
with a different gesture and it manipulates the word 
list 744 rather than the stroke group list 712. 

FIGURES 5A and SB show a flowchart which 
illustrates the operation of the insert character dia- 
log 826 for inserting a character in an existing 
displayed string, which may be either handwritten 
or recognized. In a step 510 of FIGURE 5A. the hit 
test primitive 620 responds to a command by the 
writingD widget 220 to locate the entry on the 
stroke element list 710 or the stroke group list 712 
whose positional data 718 corresponds to the dis- 
play coordinates of the gesture. 

In a step 512, the writingR widget 218 invokes 
the database primitives 614 to adjust the positional 
data 718 of the unrecognized stroke group entries 
720 or recognized stroke group entries 726 so as 
to produce a gap where a new character may be 
inserted. The database 212 then calls back the 
writjngA widget 210 in step 514 to display the 
strokes or data with the insertion gap. Execution of 
the insert character dialog 826 is then suspended 
(but not deactivated) in a step 516 to receive a 
stroke from the user 128. 

Execution returns to the writing widgets 210. 
218 and 220 at the step 312 of FIGURE 3A. The 
writing widgets 210. 218 and 220 process the 
stroke entered by the user 128, and then execute 
the insert character dialog 826 in the step 390 of 
FIGURE 3E. 

Upon being executed with the new stroke, the 
insert character dialog 826 determines (in a step 
518 of FIGURE 5A) whether there has been a 
closure event If not. the dialog is again suspended 
to receive a stroke. Once there has been a closure 
event, then the insert character dialog 826 carries 
out steps 520 through 536 of FIGURES 5A and 5B 
to recognize the character entered, add it to a word 
Nst entry 752, and display it in place of the entered 
strokes. Steps 520 through 534 are identical to 
steps 350 through 364 of FIGURES 3B and 3C. 
which are described above. Finally, in a step 536, 
the insert character dialog 826 is deactivated. 

The insert word dialog 832 operates in a simi- 
lar manner, except that It is invoked with a different 
gesture and it operates on the word level rather 
than the character level. 

The external dialogs 812 are readily added, 
removed or modified by editing the dialog file. No 
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modification of the source cod© of the TAPI 110 
itself is required. FIGURE 9 shows a flowchart of 
the method for creating a new dialog. First, in a 
step 910. the software application developer de- 
fines a new invocation stroke (he., the gesture with 6 
which the user 128 will invoke the new dialog). 
Second, in a step 912, the application developer 
trains the gesture recognizer 125 to recognize the 
new gesture and to return a new dialog identifica- 
tion code to the TAPI 110 when it Is recognized. to 
Training may consist of capturing several copies of 
the invocation stroke data and inserting them in the 
gesture prototype data set 146 of the gesture re- 
cognizer 125. 

Finally, in a step 914, the developer defines the re 
new dialog in the dialog file. The dialog definition 
consists of "nouns" and -verbs" which act upon 
the nouns. The nouns correspond to properties of 
the dialog strokes such as the invocation stroke 
and any subsequent strokes received by the dia- 20 
log. The subsequent strokes are input strokes to 
the dialog, including gestures and characters. 

The properties further include any stroke ele- 
ment entries 714, stroke group entries 720 or word 
list entries 752 identified by the invocation stroke. 25 
For example, in the case of the delete character 
dialog 824. the invocation stroke identifies the 
stroke element entry 714 of a stroke to delete or 
the stroke group entry 720 of a character to delete. 

The verbs are database primitives 614 or 30 
macros for carrying out the action of the dialog. 
Macros comprise database primitives 614 or other 
dialogs. Both the macros and the other dialogs are 
contained in the dialog file. 

To remove a dialog, the software application 35 
developer removes the dialog entry from the dialog 
file and removes the associated gesture from the 
gesture prototype data set 146. To modify a dialog, 
the software application developer edits the dialog 
file. 40 

Mote that if the software application developer 
had access to the source code of the TAP1 110, he 
or she could create an external dialog 812 by 
incorporating its features in the software of the 
gesture recognizer 144. The software application 45 
developer could delete an external dialog 812 in- 
corporated into the gesture recognizer 144 by re- 
moving it from the gesture recognizer 144. and he 
or she could modify it by modifying the gesture 
recognizer 144. so 

Conclusion 

While the invention has been particularly 
shown and described with reference to preferred 55 
embodiments thereof, it will be understood by 
those skilled in the art that the foregoing and other 
changes in form and details may be made therein 



without departing from the spirit and scope of the 
invention. 

Claims 

1. A method in a tablet computer for disam- 
biguating a handwriting input stream generated 
by a stylus-based user interface comprising 
the steps of: 

(a) receiving x-y data representative of a 
stroke made on the stylus-based user inter- 
face; 

(b) converting said x-y data to positional 
data, wherein said positional data indicates 
the relative position of said x-y data on a 
display device; 



(c) inserting said positional data into a 
stroke element entry of a database; and 




2. The method of claim 1. further comprising the 

(a) writing said database to a file upon a 
request to suspend said application pro- 

(b) restoring said database from said file 
upon a request to resume said application 

3. The method of claim 1 or 2, further comprising 
the steps of: 

(a) routing said positional data to a gesture 

(b) receiving a dialog identification code 
from said gesture recognizer; and 

(c) executing a dialog indicated by said 
dialog identification code. 

4. The method of claim 1,2 or 3, further compris- 
ing the steps of: 

(a) repeating steps (a) and (b) of claim 1 to 
generate subsequent positional data; and 

(b) routing said subsequent positional data 
to said dialog. 

5. The method according to any one of the pre- 
ceding claims, further comprising the steps of: 

(a) defining an invocation stroke and a dia- 
log identification code in said gesture re- 
cognizer; 

(b) inserting in a prototype data set of said 
gesture recognizer gesture positional data 
representative of said invocation stroke; and 

(c) inserting in a dialog file database ma- 
nipulations to be performed when said ges- 
ture recognizer matches said positional data 
of step (a) of claim 3 to said gesture posi- 



10 



19 



EP0565 872A2 



20 



Bonal data. 

6. The method according to any one of the pre- 
ceding claims, further comprising the steps of: 

(a) deleting an invocation stroke and a dia- s 
log identification code from said gesture 
recognizer; 

(b) deleting from a prototype data set of 
said gesture recognizer gesture positional 
date representative of said invocation io 
stroke: and 

(c) deleting from a dialog file database ma- 
nipulations to be performed when said ges- 
ture recognizer matches said positional data 

of step (a) of claim 3 to said gesture posi- , 5 
tiortal data. 

7. The method according to any one of the pre- 
ceding claims, further comprising the step of 
modifying a definition of an invocation stroke so 
and a dialog identification code in said gesture 

a The method according to any one of the pre- 
ceding claims, further comprising the step of 25 
modifying in a prototype data set of said ges- 
ture recognizer gesture positional data repre- 
sentative of said Invocation stroke. 

9. The method according to any one of the pre- 30 
ceding claims, further comprising the step of 
modifying in a dialog file database manipula- 
tions to be performed when said gesture re- 
cognizer matches said positional data of step 
(a) of claim 3 and said gesture positional data. 35 

10. The method according to any one of the pre- 
ceding cjaims,further comprising the steps of: 

(a) repeating steps (a) - (c) of claim 1 until a 
closure event is received; « 



(b) grouping said stroke element entries so 
as to generate a stroke group entry of said 

(c) routing sakf positional data of said stroke 
element entries to a character recognizer; 

(d) receiving a character result from said 
character recognizer; and 

(e) inserting said character result into said 
stroke group entry. 

11. The method according to any one of the pre- 
ceding claims, further comprising the steps oft 

(a) determining whether there is a word 
entry in said database with which a char- 
acter comprising said character result is as- 
sociated; 

(b) inserting said character result into said 
word entry if said word entry exists; 



(c) creating a new word entry of said 
database if said word entry does not exist; 
and 

(d) inserting said character result into said 
new word entry if said word entry does not 

12. The method according to any one of the pre- 
ceding claims, further comprising the steps of: 
(a) inserting said positional data into a re- 
cognizer input structure; 



(b) routing said recognizer input structure to 
said recognizer; 




structure a result structure; and 

(e) inserting said result structure into said 

database. 



13. An apparatus in a tablet computer for disam- 
biguating a handwriting input stream generated 
by a stylus-based user interface comprising: 

(a) means for receiving x-y data representa- 
tive of a stroke made on the stylus-based 
user interface; 

(b) means for converting said x-y data to 
positional data, wherein said positional data 
indicates the relative position of said x-y 
data on a display device; 

(c) means for inserting said positional data 
into a stroke element entry of a database; 
and 

(d) means for notifying an application pro- 
gram that said database has been updated. 

14. The apparatus of claim 13, further comprising: 

(a) means for writing sakf database to a file 
upon a request to suspend said application 
program; and 

(b) means for restoring said database from 
said file upon a request to resume said 
application program. 

45 15. The apparatus of claim 13 or 14, further com- 
prising: 

(a) means for routing said positional data to 
a gesture recognizer; 

(b) means for receiving a dialog identiffca- 
50 Hon code from said gesture recognizer; and 

(c) means for executing a dialog indicated 
by said dialog identification code. 

16. The apparatus of claim 15, further comprising: 
56 (a) means for receiving subsequent x-y data 

representative of a subsequent stroke made 
on the stylus-based user interface; 
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(b) means for converting said subsequent x- 
y data to subsequent positional data, 
wherein said subsequent positional data in- 
dicates the relative position of said subse- 
quent x-y data on said display device; and s 

(c) means for routing said positional data to 
said dialog. 

17. The apparatus of claim 15 or 16. further com- 
prising: 10 

(a) means for defining an invocation stroke 
and a dialog identification code in said ges- 
ture recognizer: 

(b) means for inserting in a prototype data 

set of said gesture recognizer gesture post- 75 
tional data representative of said invocation 
stroke; and 

(c) means for inserting in 8 dialog fife 
database manipulations to be performed 
when said gesture recognizer matches said so 
positional data of said routing means of 
claim 15 to said gesture positional data. 

18. The apparatus of claim 15. 16 or 17 further 
comprising: 2s 

(a) means for deleting an invocation stroke 
and a dialog identification code from said 
gesture recognizer; 

(b) means for deleting from a prototype data 

set of said gesture recognizer gesture posi- 30 
tional data representative of said invocation 
stroke; and 

(c) means for deleting from a dialog file 
database manipulations to be performed 
when said gesture recognizer matches said 35 
positional data of said routing means of 
claim 15 to said gesture positional data. 

19. The apparatus according to any one of the 
claims 13 to 18, further comprising means for to 
modifying a definition of an invocation stroke 

and a dialog identification code in said gesture 
recognizer. 

20. The apparatus according to any one of the 45 
claims 13 to 19. further comprising means for 
modifying in a prototype data set of said ges- 
ture recognizer gesture positional data repre- 
sentative of said invocation stroke. 

so 

21. The apparatus according to any one of the 
claims 13 to 20. further comprising means for 
modifying in a dialog file database manipula- 
tions to be performed when said gesture re- 
cognizer matches said positional data of said 55 
routing means of claim 15 to said gesture 
positional data. 



22. The apparatus according to any one of the 
claims 13 to 21. further comprising: 

(a) means for repeatedly invoking said re- 
ceiving means of claim 13, said converting 
means of claim 13, and said inserting 
means of claim 13 until a closure event is 
received; 

(b) means for grouping said stroke element 
entries so as to generate a stroke group 
entry of said database; 

(c) means for routing said positional data of 
said stroke element entries to a character 
recognizer; 

(d) means for receiving a character result 
from said character recognizer, and 

(e) means for inserting said character result 
into said stroke group entry. 

23. The apparatus according to any one of the 
claims 13 to 22. further comprising: 

(a) means for determining whether there is 
a word entry in said database with which a 
character comprising said character result is 
associated: 

(b) means for inserting said character result 
into said word entry if said word entry ex- 
ists: 

(c) means for creating a new word entry of 
said database if said word entry does not 
exist and 

(d) means for inserting said result into said 
new word entry if said word entry does not 

exist. 

24. The apparatus according to any one of the 
claims 13 to 23, further comprising: 

(a) means for receiving said positional data; 

(b) means for inserting said positional data 
into a recognizer input structure; 

(c) means for routing said recognizer input 
structure to said recognizer; 

(d) means for receiving a recognizer output 
structure from said recognizer, 

(e) means for generating from said recog- 
nizer output structure a result structure; and 

(f) means for inserting said result structure 
into said database. 
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